Mitigating the Position Bias of Transformer Models in Passage Re-ranking

Supervised machine learning models and their evaluation strongly depends on the quality of underlying dataset. When we search for a relevant piece information it may appear anywhere in given passage. However, observe bias position correct answer text two popular Question Answering datasets used passage re-ranking. The excessive favoring earlier positions inside passages is an unwanted artefact. This leads to three common Transformer-based re-ranking ignore parts unseen passages. More concerningly, as set taken from same biased distribution, overfitting that overestimate true effectiveness. In this work analyze datasets, contextualized representations, effect retrieval results. We propose debiasing method datasets. Our results show model trained position-biased dataset exhibits significant decrease effectiveness when evaluated debiased demonstrate by mitigating bias, are equally effective dataset, well more transfer-learning setting between differently